This document covers my initial exploration of the Yelp Dataset Challenge data. This dataset from the online review site Yelp, and is part of a data anlysis competition, now in its fifth run. The data contains (from the Yelp website):

Initialisation

To start, I’ll import my libraries, include files, and set globals variables. After that, I’ll load in the data (from cache if available).

Now that the data is loaded, I’ll quickly verify the Yelp statements about the dataset size, and take a quick look at the structure of the y.business data frame.

##         Stat    Rows
## 1    Reviews 2225213
## 2       Tips  591864
## 3      Users  552339
## 4 Businesses   77445
## 'data.frame':    77445 obs. of  15 variables:
##  $ business_id  : chr  "5UmKMjUEUNdYWqANhGckJw" "UsFtqoBl7naz8AVUBZMjQQ" "3eu6MEFlq2Dg7bQh8QbdOg" "cE27W9VPgO88Qxe4ol6y_g" ...
##  $ full_address : chr  "4734 Lebanon Church Rd\nDravosburg, PA 15034" "202 McClure St\nDravosburg, PA 15034" "1 Ravine St\nDravosburg, PA 15034" "1530 Hamilton Rd\nBethel Park, PA 15234" ...
##  $ hours        :'data.frame':   77445 obs. of  7 variables:
##  $ open         : logi  TRUE TRUE TRUE FALSE TRUE TRUE ...
##  $ categories   :List of 77445
##   .. [list output truncated]
##  $ city         : chr  "Dravosburg" "Dravosburg" "Dravosburg" "Bethel Park" ...
##  $ review_count : int  4 4 3 5 5 20 3 21 7 4 ...
##  $ name         : chr  "Mr Hoagie" "Clancy's Pub" "Joe Cislo's Auto" "Cool Springs Golf Center" ...
##  $ neighborhoods:List of 77445
##   .. [list output truncated]
##  $ longitude    : num  -79.9 -79.9 -79.9 -80 -80.1 ...
##  $ state        : chr  "PA" "PA" "PA" "PA" ...
##  $ stars        : num  4.5 3.5 5 2.5 2.5 5 2.5 4 2.5 4 ...
##  $ latitude     : num  40.4 40.4 40.4 40.4 40.4 ...
##  $ attributes   :'data.frame':   77445 obs. of  36 variables:
##  $ type         : chr  "business" "business" "business" "business" ...

Geography

I know that the businesses are located in 10 distinct areas. I have their names, and the latitude/longitude are in the y.business object. I’ll use k-means clustering to label the groups, which will allow for some comparative geo-analysis. I have the list of actual locations which the data should be from, so i will seed the kmeans with the lat/long of those ten locations.

##                       
##                                 AL    AR    AZ    BW    CA   EDH   ELN
##   Charlotte, NC            0     0     0     0     0     0     0     0
##   Edinburgh, UK            0     0     0     0     0     0  3206    12
##   Karlsruhe, Germany       0     0     0     0  1048     0     0     0
##   Las Vegas, NV            0     0     0     1     0     2     0     0
##   Madison, WI              0     0     0     0     0     0     0     0
##   Montreal, Canada         1     0     0     0     0     1     0     0
##   Phoenix, AZ              0     1     1 32614     0     0     0     0
##   Pittsburgh, PA           0     0     0     0     0     0     0     0
##   Urbana-Champaign, IL     0     0     0     0     0     0     0     0
##   Waterloo, Canada         0     0     0     0     0     0     0     0
##                       
##                          FIF   HAM    IL   KHL    MA   MLN    MN    NC
##   Charlotte, NC            0     0     0     0     0     0     0  6162
##   Edinburgh, UK            5     1     0     1     0   147     0     0
##   Karlsruhe, Germany       0     0     0     0     0     0     0     0
##   Las Vegas, NV            0     0     0     0     0     0     0     0
##   Madison, WI              0     0     0     0     1     0     1     0
##   Montreal, Canada         0     0     0     0     0     0     0     0
##   Phoenix, AZ              0     0     0     0     0     0     0     0
##   Pittsburgh, PA           0     0     0     0     0     0     0     0
##   Urbana-Champaign, IL     0     0   737     0     0     0     0     0
##   Waterloo, Canada         0     0     0     0     0     0     0     0
##                       
##                           NM   NTH    NV    NW    ON    OR    PA    QC
##   Charlotte, NC            0     0     0     0     0     0     0     0
##   Edinburgh, UK            0     1     0     0     0     0     0     0
##   Karlsruhe, Germany       0     0     0     1     0     0     0     0
##   Las Vegas, NV            1     0 21233     0     0     1     0     0
##   Madison, WI              0     0     0     0     0     0     0     0
##   Montreal, Canada         0     0     0     0     0     0     0  4942
##   Phoenix, AZ              0     0     0     0     0     0     0     0
##   Pittsburgh, PA           0     0     0     0     0     0  3754     0
##   Urbana-Champaign, IL     0     0     0     0     0     0     0     0
##   Waterloo, Canada         0     0     0     0   474     0     0     0
##                       
##                           RP    SC   SCB    TX    WI
##   Charlotte, NC            0   271     0     1     0
##   Edinburgh, UK            0     0     3     0     0
##   Karlsruhe, Germany      18     0     0     0     0
##   Las Vegas, NV            0     0     0     1     0
##   Madison, WI              0     0     0     0  2802
##   Montreal, Canada         0     0     0     0     0
##   Phoenix, AZ              0     0     0     0     0
##   Pittsburgh, PA           0     0     0     0     0
##   Urbana-Champaign, IL     0     0     0     0     0
##   Waterloo, Canada         0     0     0     0     0

These results show a good grouping, where each state is entirely represented by one cluster, with the exception of a few erronously identified businesses. For instance, ‘Cavalia’ in Montreal (based on it’s latitude/longitude), has been given the address details of Cavalia in Burbank California.

## [1] "Outremont\nBurbank, CA 91502"

I’ll do a pair of map plots - one for the US, and one for Europe - to show the distribution of the business, coloured by the cluster.

The maps show that the clustering worked nicely, and also shows the geographic spread of the businesses in the dataset. Despite the scale of the USA map, you can see something of the dispersal of businesses within each cluster - the Phoenix AZ cluster is the largest of the groups on the map, while the Urbana-Champaign IL group is smallest.

I’ll add a distance to the cluster center too - I can then do some analysis of the area covered by each cluster This process uses the Haversine formula for distance on a sphere, given two pairs of coordinates in radians. The resulting figures are in kilometres. **************************************** TODO - actually get a distance, more useful for human analysis/understanding

## Source: local data frame [10 x 3]
## 
##                location        mn        sd
##                   (chr)     (dbl)     (dbl)
## 1         Charlotte, NC 10.192273  6.152191
## 2         Edinburgh, UK  1.727740  2.023256
## 3    Karlsruhe, Germany  3.165571  3.220315
## 4         Las Vegas, NV  9.040471  5.505476
## 5           Madison, WI  6.605847  4.517044
## 6      Montreal, Canada  4.859244  4.337261
## 7           Phoenix, AZ 19.680806 12.066621
## 8        Pittsburgh, PA  4.628722  2.861922
## 9  Urbana-Champaign, IL  1.743124  1.250232
## 10     Waterloo, Canada  3.740925  2.766424

This shows that Urbana-Champaign has the tightest group of businesses (low mean and standard deviation), while Phoenix has the most dispersed, as seen on the map plots.

## Warning: Removed 991 rows containing missing values (geom_point).
## Warning: Removed 4 rows containing missing values (geom_point).

**************************************** TODO MORE GEO STUFF!!!!!
Maybe a multivariate plot of some kind? Deeper analysis of a single city?

Business Categories

I’d also like to get a primary category for each business - this will also simplify analysis. The primary category list is taken from https://www.yelp.com/developers/documentation/v2/all_category_list.

## [1] 346

There are 346 combinations of primary categories! This makes any specific comparison of primary categories tricky, as businesses could belong to several primary categories. Taking an slightly different tack led me to think more deeply about the categories and review counts. Do businesses with more categories have more reviews and/or higher ratings?

There isn’t a clear link between the ratings and number of categories, but there appears to be a clear link between categories and number of reviews

*********************************** Need to change this to two plots

Using a log plot of reviews shows there are actually few businesses with very high numbers of reviews. The point density shows the bulk of businesses have between 2 and 5 categories, and less than 100 reviews

Checkins

I had hoped to perform more category-based analysis, but with multiple primary categories, I would have to sift the combinations and derive so kind of hierarchy to determine a single primary for each business. Instead, I chose to look into a different view of the data - checkins. This shows when businesses are visited, by week-hour. The y.checkin data frame contains my core data - I’ll add some of the columns from y.business to provide a more useful single frame. I will use heatmaps to visualise this data.

Here we can see a clear pattern in restaurant checkins - more frequent at lunchtimes and evenings, and also on Friday and Saturdays. The nightlife category should clearly show a similar pattern…

A similar pattern, but with more emphasis on the weekend evenings. Shopping patterns show that Saturday is the most popular day across the ten locations. Sunday trading shows an interesting pattern, particularly in Karlsruhe where there is almost no activity - Germany has strict Sunday trading laws, and most shops will be closed.

Users

****************** TODO Expand on the text Initial look at the user data Improve - remove extreme outliers, log the scales, add the elite.years as a colour This plot appears to show a mild correlation between reviews and fans, but the data is still far too compressed. I’ll remove the low-fan users, and see what impact that has. Now we can see the correlation between review_count and fans more clearly - and additionally we can see a peculiar grouping of high fan, low review_count users. This is a strange grouping of outliers, particularly as they seem to be clustered around 70 fans. I could speculate many reasons for this, but without sufficient supporting data I can’t make any confident statements. I do, however, consider them to be an anomaly, and will exclude them from further analysis (fans>60, review_count<30).

Look at the correlations in the user stats:

We can see that there is no correlation between average_stars and any of the other values, but there is a strong correlation between the votes columns, particularly votes.funny and votes.useful.

****** TODO Further user analysis - 2-3 more vis

Final Plots and Summary

****** TODO Select my 3 plots, make them perfect. Suggest: ****** TODO USA map, heat map of religion in Phoenix, one other TBD

Reflections

******* TODO 200 words Large multi-layered dataset. Primarily categorical/ text based. Well suited for a NLP/prediction project.